Method and Apparatus for Implementing Automatic Protection Switching Functionality in a Distributed Processor Data Router

ABSTRACT

An automated-protection-switching (APS) software suite for distribution over multiple processors of a distributed processor router has an APS server module running on a first one of the multiple processors for managing communication and distributing configuration and state information and APS client modules running on second ones of the multiple processors, the APS client modules for monitoring interface state information, reporting to the APS server application, and for negotiating with other APS client modules. The software is characterized in that APS interface relocation from a primary interface to a backup interface is performed through direct communication between the APS client modules running on the processors supporting the involved interfaces.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention is in the fields of data-packet-processing andforwarding packets over a data-packet-network, and pertains particularlyto methods and apparatus for enabling ASP function in a router usingmultiple processors and distributed functionality.

2. Description of the State of the Art

The field of data communication has grown with the pace of newdata-routing equipment, switches, and techniques that have enabled alltypes of data to be transmitted over wide area networks (WAN) faster andmore reliably. Manufacturers are competing to introduce faster datarouters and better methods for synchronizing and integratingstate-of-the-art equipment provided by a variety of competitors.

Integrating various types of proprietary data routing equipment fornetwork communication involves, among other things, dealing withdifferent rates of data processing and transfer in terms of interfacesand interfacing data network protocols between disparate data networks.

One of the more recent advances in data routing over large WANs such asthe well-known Internet network, for example, is the implementation ofsynchronized optical network or SONET protocol and equipment. SONET is astandardized protocol implemented along with specialized networkequipment to create a special network that allows multiple data linestransmitting data at different rates and formats to be multiplexed overa single optical carrier typically at a higher speed through a SONETportion or section of the network.

A simplified example of SONET might involve multiplexing, for example, aT1 line transferring data at 1.54 Mbits/s, a T3 line transferring dataat 44.736 Mbits/s, and a E1 line transferring data at 2.048 Mbits/sec,onto a single fiber-optic cable after processing the separate streamsthrough a SONET Multiplexor system. The result is a single combinedstream transferred at a higher rate, typically 51.48 Mbits/s associatedwith fiber-optics technology. At the other end of the SONET fiber-opticcable, the data streams are de-multiplexed and resume their originalformats, characteristics and transfer rates. SONET is a midstreamsolution for getting data across the network between oftentimesdisparate vendor interfaces. Data may travel in this fashion throughestablished SONET network paths until the streams are terminated as faras SONET format is concerned and converted back to user formats.

Much detailed information on the SONET structure and parameters ispublicly available, therefore the publicly-available detail will not beprovided here except to say that the application of SONET enablessubscribers to configure their router interfaces so that a multiple ofsuch interfaces on a single router become dedicated SONET lines.

Another convention in the art of data transfer and switching is termedautomatic protection switching (APS). APS comprises a protocol andsoftware that enables a plurality, typically an aggregated group, ofprimary lines egressing from a router to be individually backed-up by asingle backup line dedicated for the purpose.

In one application, APS is used to provide some redundancy for a groupof dedicated SONET line interfaces of a router. For example, a group ofseparate router interfaces on the egress side of the router that areconnected to SONET equipment is backed-up by a dedicated interface(backup line). Typically, for APS switchover from a primary terminal toa backup to be successful, it must occur within 50 milliseconds (ms),which is a standard set within the protocol. Also, of course, all of theprimary terminal parameters, such as data transport protocols and stateinformation must be identically implemented at the backup terminal toobtain a successful handshake at the other end of the communicationpath. APS fault protection is transparent to the other communicatingparty or system. Moreover, in the case of more than one primary SONETline requiring or requesting backup at a same time, a priority schememay be implemented to enable priority selection of a line for backup or“relocation” as it is termed in the art.

A data router known to the inventor uses a distributive processingarchitecture to process data. The router is termed a terabit networkrouter (TNR) developed to improve data routing efficiency in general,and to enable users to scale router capacity easily. The distributedprocessing components include line cards that interface between theinternal router domain and the external connected network, fabric cardsand interconnections that comprise an internal data packet routingnetwork within the router itself, and control cards that provide controlroutines, messaging, and in some cases special packet processing duties.

Each card in a TNR typically has a dedicated, on-board processor andmemory. Each card also typically has a plurality of physical interfacescomprising ingress/egress ports for transferring data. The presentinvention in preferred embodiments primarily focuses on the use of SONETAPS protection on a distributed processor router, and focuses moreparticularly on a novel implementation and use of APS software in adistributed processor system.

Prior art routers employing one, or at most a few processors, can useconventional APS software because all of the processing involved happenson one processor for all of the SONET lines, and there is a designatedbackup, as will be described further below with reference to a prior-artexample of FIG. 1. It was described above that the standard of 50 ms forswitchover must be adhered to for APS to be successful. If a switchoverprocess times out before completion, the communication path involvedwill not exhibit a successful handshake and transmission will fail. Adistributed processor system cannot perform a switchover within therequired time period using standard APS software. Therefore, what isclearly needed is a method and apparatus (software) that will allow APSto be successfully practiced on a multi-processor router.

SUMMARY OF THE INVENTION

In a preferred embodiment of the invention anautomated-protection-switching (APS) software suite for distributionover multiple processors of a distributed processor router is provided,comprising an APS server module running on a first one of the multipleprocessors for managing communication and distributing configuration andstate information, and APS client modules running on second ones of themultiple processors, the APS client modules for monitoring interfacestate information, reporting to the APS server application, and fornegotiating with other APS client modules. The software is characterizedin that APS interface relocation from a primary interface to a backupinterface is performed through direct communication between the APSclient modules running on the processors supporting the involvedinterfaces.

In one preferred embodiment the distributed processor router isconnected to and operating on a data-packet-network, which may be theInternet network. In preferred embodiments the novel APS software isimplemented to protect the integrity of a plurality of primaryinterfaces of the router by enabling backup of individual ones of theinterfaces at any given time during router operation. In some cases theplurality of primary interfaces comprise an APS grouping of interfacesconnected to a SONET network.

In preferred embodiments configuration and state information generic toa primary interface for relocation is mirrored to the client supportingthe backup interface for the purpose of initializing and activating thebackup interface to function as the primary interface. Typically thedistributed processors communicate with each other through a network offabric cards implemented within the router. Also in preferredembodiments all communication exchanges between the distributed APScomponents follow a message sequence scheme wherein each request andresponse has a sequence number.

In preferred embodiments of the invention interface relocation isinitiated by an APS client module after detecting an event requiringrelocation at the primary interface to be relocated. In variousscenarios the APS grouping of interfaces may be physically supported onone processor, or distributed to and physically supported by multipleprocessors.

In an alternative aspect of the invention a distributed processor routeris provided, comprising a plurality of communicating processorssupporting a plurality of communication interfaces, an APS server modulerunning on a first one of the plurality of processors for managingcommunication and distributing configuration and state information, andAPS client modules running on second ones of the multiple processors,the APS client modules for monitoring interface state information,reporting to the APS server module, and for negotiating with other APSclient modules. The router is characterized in that APS interfacerelocation from a primary interface to a backup interface is performedthrough direct communication between the APS client modules running onthe processors supporting the involved interfaces

In preferred embodiments of the router the data-packet-network is theInternet network Also in some preferred embodiments the plurality ofprimary router interfaces comprise an APS grouping of interfacesconnected to a SONET network. Also in preferred embodiments the APSsoftware suite includes a server application, a server-clientapplication, and a client module. In some cases the server applicationruns on a control card, and the server-client application as well as theclient module run on a line card.

In operation of the router indication of an event may be an APS signalreceived through the target interface on the backup processor, and thereceived APS signal may indicate one of failure or severe degradation ofthe target interface. The received APS signal may indicate anadministrative request for interface relocation.

In preferred embodiments configuration and state information generic toa targeted interface for relocation is mirrored to the backup routerinterface for the purpose of initializing and activating the backupinterface to function as the primary interface. The distributedprocessors typically communicate with each other through a network offabric cards implemented within the router.

Still in preferred embodiments of distributed processor router allcommunication exchanges between the distributed APS components follow amessage sequence scheme wherein each request and response has a sequencenumber. In some cases the primary and backup processors may comprise thesame processor.

In still another aspect of the invention a method for relocating aprimary router interface to a designated backup router interface usingan APS suite distributed over multiple processors of a distributedprocessor data router is provided, comprising the steps of (a) mirroringcurrent configuration and state information of the primary routerinterface to the processor supporting the designated backup routerinterface; (b) receiving indication of a requirement to initiate an APSswitchover; (c) determining if the backup router interface is available;and (d) activating the designated backup interface using the mirroredconfiguration and state information.

In some embodiments there is an additional step for reporting anychanged route results to a task manager responsible for distributingupdated route tables to processors, and a further step for updating aforwarding data base according to a switchover made. IN preferredembodiments the distributed processor data router is connected to andoperating on a data-packet-network at the time of interface relocation,and the data- packet-network may be the Internet network. The primaryrouter interface may be a part of a group of interfaces connected to aSONET network.

In some embodiments, in step (b) the indication is received at theprimary interface. In others, in step (b), the indication is received atthe backup interface. The indication may be of the form of anadministrative request.

In some embodiments the backup interface partly depends on a prioritystate of the interface requiring backup. In some cases, in step (c) thebackup interface is physically located on a processor separate from thatof the primary router interface. In step (a) the configuration and stateinformation may be selected from a table of such sets of informationstored on the processor hosting the backup router interface.

BRIEF DESCRIPTION OF THE DRAWING FIGURES

FIG. 1 is a network diagram illustrating data routers communicatingthrough the SONET network using APS protocol according to prior art.

FIG. 2 is a network diagram illustrating a distributed processor routercommunicating through the SONET network using APS protocol according toan embodiment of the present invention.

FIG. 3 is a block diagram illustrating APS components of a distributedAPS software suite according to an embodiment of present invention.

FIG. 4 is a block diagram illustrating components of aninterface-configuration-replications-set according to an embodiment ofpresent invention.

FIG. 5 is a process flow diagram illustrating steps for initiating anAPS switchover according to an embodiment of present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

FIG. 1 is a network diagram illustrating data routers 107 and 108communicating through a SONET network 102 using APS protocol accordingto prior art. SONET network 102 is analogous to any SONET networkoperating SONET protocols and equipment. In this simple example, SONETnetwork 102 comprises an add/drop multiplexor (ADM) 104, and an ADM 105,the latter connected to the former via a fiber optics line 103. Thedescribed equipment connection represents SONET network 102. It will beappreciated that there will likely be many sections comprising SONETlines and equipment included within SONET network 102. Only two piecesof SONET equipment and a single fiber optics line connecting the two aredeemed sufficient for describing the present invention.

SONET 102 is contained within a wide-area-network (WAN) 100, which isthe Internet network in this example. Data router 108 is analogous to astandard prior-art data router as may be known in the art for routingdata through a data-packet network. Data router 108 has a CPU 112therein adapted to perform all packet processing of the router. Datarouter 108 is configured in this example for SONET operation. Forexample, an APS group of interfaces 109 is provided as interfaces thathave connection through Internet 100 to ADM 104 within SONET 102. APSgroup 109 comprises 4 primary and active interfaces and one APS backupinterface 110 (logically illustrated dotted line).

For the purpose of simplicity, it will be assumed that each individualAPS line including backup 110 of APS group 109 comprises one egress portof router 104 that is configured for SONET and APS protection. Also forsimplicity, ingress of router 108 is illustrated as 4 double-arrowsimplying 4 separate ports. In actual practice, all ports on router 108are ingress/egress ports and there may be many more than 4 portsprovided on each side of router 108 of which many may be active andSONET/APS configured. APS backup 110 of group 109 is typically an idleinterface until backup services are required.

At the opposite end of SONET 102, router 107 is adapted identicallyaccording to the description provided with reference to router 108. Theidentical implementation is exemplary only, and router 107 could well beadapted differently than router 108. For example, router 107 has a CPU113 therein adapted to perform packet processing of router 107. An APSgroup 106 comprises the primary and active lines and one idle APS backupline 111. In this prior-art example both routers 108 and 107 areidentically adapted for APS/SONET data transfer. Therefore, it may beassumed in this example that both routers 108 and 107 have standard APSsoftware installed therein.

If router 108 detects a failure in any one or more of the primary SONETlines within APS group 109, then APS backup line 110 will be configuredto take over for the highest priority line that has failed. APS backup110 may also be configured for one of the primary lines based onadministrative request. It is important to note here that only one ofthe primary lines may be relocated to interface 110 at any given time.Because router 108 utilizes a single CPU 112, all of the state andconfiguration information of all of the primary protected lines isavailable locally. As a result, interface 110 may typically beconfigured easily within the 50-millisecond window required by APSprotocol.

While APS backup 110 is functioning as one of the primary lines, theinterface that is being backed up is suspended from operation until theproblem with that line no longer exists, or an administrator requests areverse switchover. When router 108 detects that a primary linecurrently relocated to interface 110 is up again, the interface isre-activated and the backup interface 110 becomes idle again.

In this prior-art example APS software is adapted to run on a singleprocessor and not on multiple processors of a same router. As describedwith reference to the background section, it is a goal of the presentinvention to provide a unique distributed APS software suite that willoperate with multiple-processor routers, wherein all of the requiredcommunication between distributed APS components can be completed toperform a switchover within the 50 millisecond time window provided byAPS protocol. The method and apparatus of the present invention isdescribed in more detail below.

FIG. 2 is a network diagram illustrating a distributed processor router201 communicating through a SONET network using APS protocol accordingto an embodiment of the present invention. In a preferred embodiment ofpresent invention data router 201 utilizes a plurality of processorswhose functions are to distribute the responsibility of packetprocessing over multiple points of processing within the router. Router201 is termed a terabit network router (TNR) by the inventor. Router 201is hereinafter referred to as TNR 201.

In this example all of the illustrated equipment is assumed to beoperating within the domain of a WAN 200 (network cloud notillustrated), which in a preferred embodiment is the Internet network.Referring now back to TNR 201, there is illustrated a plurality ofprocessing components termed cards by the inventor. The three types ofprocessing cards illustrated in varying detail in this example andprovided within TNR 201 are line cards labeled herein as LC, controlcards labeled herein as CC and GMCC (global-master-control-card), andfabric cards illustrated logically herein as a network cloud 206 labeledFabric.

For the purpose of general description, line cards in the TNR generallyrepresent interfaces between internal components of TNR 201 and theexternal network to which it is connected. Fabric 206 represent aplurality of cards interconnected to form an internal data network ofoperating nodes within TNR 201 over which packets are routed fromingress to egress and communications are routed between various types ofcards. Control cards represent the processing components that controlcommunication and other processes with respect to other cards in theoperation and synchronization of TNR 201 as a whole.

In this example there are 2 line cards represented within TNR 201 as LC205 and LC 204. There are 2 control cards represented within TNR 201 inthis example as CC 202 and GMCC 203. GMCC 203 is designated as a masteror primary control card in this specific example. Fabric 206 comprisesall the fabric cards within TNR 201, and all of the interconnectingcircuitry in the fabric. Data traffic addressed to any IP destinationreachable through Internet 200 ingresses TNR 201 through a line card, isrouted typically through fabric 206, and egresses TNR 201 through a linecard as well. It is noted herein that one line card may function as aningress/egress for a same data stream. Both the line cards and thecontrol cards communicate with each other through fabric 206.

In this example each illustrated line card (205,204) and control card(202,203) has a plurality of physical ports. In this embodiment eachcard has eight physical ports, however there may be many more or fewerports on each card without departing from the spirit and scope of thepresent invention. The ports for the line cards are externally-facingports for communication with other network nodes, while the ports of acontrol card are for internal communication. TNR 201 has connection, inthis example, to SONET-compatible network 102 through a dedicated APSgroup 207 as was described with reference to FIG. 1 above, with theexception that the primary line interfaces of group 207, alsoillustrated herein as lines A, B, C, and D are distributed over two linecards, namely card 205 and card 204 in this example. This is notrequired to practice the present invention but is illustrated so fordiscussion purposes only. In this example there are 4 primary SONETlines, two of which (A, B) connect to LC 205 and the remainder (C, D)connect to LC 204. An APS backup line 215, illustrated herein as adotted line labeled E within APS group 207 is connected to LC 204. Usingthis configuration it may be described that in terms of APS group 207 LC205 is a primary line card while LC 204 is a backup line card. Thisdesignation is made because of the location of APS backup line 215.

Although it is not specifically illustrated in this example, eachphysical port on LC 205 and 204 that as connection to SONET network 102is configured according to a protocol known to the inventor aspath-over-SONET (POS). POS is a SONET protocol, and a port configured torun the protocol is termed a POS device by the inventor. Hereinafter inthis specification, physical ports adapted for SONET communication maybe described as POS interfaces. It is noted that in actual practicethere may be 14 primary POS interfaces and one backup POS interfacemaking up a single APS group. The inventor illustrates only 4 primaryand 1 backup interface in this example for simplification of descriptiononly.

It is also noted that a single POS interface comprising a SONET line mayconsist of multiple data channels termed SONET paths by the inventor.Each SONET path is dedicated to transmit data from a specific data flow.Therefore, multiple data flows forwarded through TNR 201 maysimultaneously use a single SONET line. In one embodiment, a SONET linemay be configured and dedicated to handle a single data flow at anygiven time. The fact that SONET protocol is used in this example in noway limits the scope of the present invention. In an alternateembodiment, physical ports on LC 205 and 204 may be configured for otherdata transfer protocols such as Ethernet and so on. The fact that SONETprotocols and equipment are used in this example represents a preferredembodiment.

GMCC 203 is identical in physical respects to CC 202 and other CCswithin TNR 201. However, in this case, GMCC 203 has a primarydesignation as the CC that will oversee APS switchover communicationassociated with APS group 207 and communicated between LCs 205 and 204.Other line cards within TNR 201 having APS/SONET primary lines and abackup line may be facilitated by some other control card adapted forthe purpose, or by CC 203 as well. In this case however, it is assumedthat GMCC 201 can facilitate all designated APS groups within TNR 201regardless of distribution order among line cards.

GMCC 203 has a memory 216 provided therein and adapted to store softwareand data. In a preferred embodiment memory 216 is a mix of volatile andnon-volatile memory. A novel APS module termed an APS interface manageris provided in distributed fashion (divided into cooperating components)on GMCC 203 as an interface manager server (IFMS) 211 and on both LCs205 and 204 as an interface manager client (IFMC) 212. IFMS 211functions as a facilitator and manager for communication between LCs 205and 204 during APS switchover activity. IFMS 211 communicates with IFMC212 through fabric 206 in the form of control messaging.

An APS client (APS CL) 213 is provided as a distributed component of thenovel APS software of the present invention and is implemented in thisexample on both LCs 205 and 204. Because LC 205 is a primary line card(PLC) and LC 204 is a backup card (BLC) in this example, APS CL 213 onLC 205 is appropriately designated as a primary APS client (PAC) and APSCL 213 on LC 204 is appropriately designated as a backup APS client(BAC). Backup designation simply refers to the fact that LC 204 supportsthe designated POS interface that provides the backup services for allof the primary lines of group 207. It is noted herein that theabove-described configurations can be changed at any time by anadministrator without having to redistribute instances of APS CL orIFMC. Each instance of APS CL is identical in capability to every otherinstance. The same is true with instances of IFMC.

SONET network 102 is analogous in every respect to the network of thesame element number described with reference to FIG. 1 above. Therefore,the equipment residing therein of the same description shall retain thesame element numbers and shall not be re-introduced in this portion ofthe description. In this example, instances of APS software 209 areprovided one on ADM 104 and one on ADM 105. A router 208 is illustratedat an end of the SONET network opposite TNR 201. Router 208 may be a TNRrouter analogous to TNR 201, in which case an illustrated instance ofAPS 217 would be implemented in a distributive fashion with the samenovel components described above with reference to TNR 201. In oneembodiment, router 208 may be a single processor router, in which caseAPS instance 217 would be of a standard prior-art form adapted forsingle processor routers.

In a preferred embodiment of the present invention, APS negotiation isperformed directly between a PLC (LC 205) in this example, and a BLC (LC204) in this example. The primary supporting role of the controlsoftware IFMS 211 running on GMCC 203 is to manage configuration anddistribute it to the appropriate line cards, as well as to mirror anyrelevant state information from a primary POS location A, B, C, or D tothe backup POS location E.

The configuration and state information of a primary POS interface thatrequires backup services is mirrored to the designated backup POSlocation. This data is described as an interface configurationreplicated set (ICRS) by the inventor. More detail about parameters ofan ICRS are provided below in this specification.

In practice of the present invention using the configuration of thisexample, we will assume that one of primary lines A, B, C, or D hasdegraded in performance to the point that backup services are required.If the failure is detected on LC 205 with one of lines A or B, then LC205 will negotiate with backup APS client 213. If however, the failureoccurs on one of lines C or D on LC 204 then the negotiation for backupoccurs only on that line card. In this example, it will be assumed thata failure is detected with line A on card 205 requiring two-cardnegotiation.

APS CL 213 is responsible for monitoring states of POS interfaces A andB and therefore detects the fail signal. APS CL notifies IFMC 212 oncard 205 of the signal.

IFMC 212 on card 205, using IFMS 211 as a relay station, notifies IFMC212 on card 204 of a request for backup services for primary POSinterface A. IFMC 212 on card 204 consults the state of POS interface Ethrough APS CL 213. If it is available (not currently backing up anotherprimary line), then a response message is sent back to IFMC 212 on card205 indicating availability. IFMS 211 is included in all communicationand response exchanges functioning to manage distribution of requiredinformation and communication management.

At this point, the appropriate configuration and state informationassociated with POS interface A is mirrored to POS interface E (backup)and implemented. This is possible because IFMC 212 on BLC 204 has all ofthe interface configuration and state information sets (ICRS) entriesfor all of the primary POS interfaces of APS group 207 stored locally.Updates to these data sets are distributed through IFMS 211 as reportedby APS CL instances 213. It is noted herein that a PLC (205) has onlylocal ICRS entries associated with the primary POS interfaces of an APSgroup that it supports, in this case, interfaces A and B of group 207.For example, an ICRS associated with POS interface A may be updatedwhile A is currently in a state of backup.

In the event of availability of services on POS interface 215 on card204, all of the configuration and state information associated with POSinterface A on card 205 is implemented at POS interface location 215.The interface is configured and activated according to the providedinformation and backup services continue until a reverse switchover isordered. In one embodiment, more than one primary may be down andrequesting backup services. In this case the POS interface having thehigher priority will be backed up. If a higher priority POS interfacerequests backup services while a lower priority interface is currentlybeing backed up then a reverse switchover may be initiated to free thebackup POS interface to service the higher priority line.

When LC 205 is unavailable global route table manager (GRTM) 210 isprovided on GMCC 203. GRTM 210 is a task manager that is responsible forkeeping an updated copy of all of the routing tables for TNR 201. In theevent of a successful POS switchover from a primary to backup POSlocation, routing tables specific to the internal network (fabric 206)of TNR 201 need to be updated. POS interface A remains in a suspendedstate while POS interface E (215) is performing backup. Because backupis being performed on a different LC, data addressed for POS egress Ashould be re-routed to ingress of the BLC, in this case, LC 204. IFMS211 notifies GRTM 210 of the requirement to perform an update anddistribution process of the updated routing tables.

A command-line-interface (CLI) system is not illustrated in thisembodiment, but may be assumed to be present, wherein the interface hasaccess to a database containing all of the APS command structure andprotocol parameters. IFMS software provides support for backend CLI APSsoftware including error check and storage configurations.

The method and apparatus of the invention enables a POS backupinterface, POS 215 in this example, to be configured and activated wellwithin the required 50 ms time window. The switchover to backup, as wellas reverse switchovers when primary lines are up again, is transparentto communicating parties on the SONET network.

FIG. 3 is a block diagram illustrating APS components of a distributedAPS software suite 300 according to an embodiment of present invention.Suite 300 is made up of distributed software modules that reside on a CCand that reside on an LC as previously described with reference to FIG.2. Not including APS software implemented at a CLI interface, thedistributed components are described below.

Implemented on a control card (lower layer labeled Control) are a GRTMmodule 301 and an IFMS module 306. These components are analogous toGRTM 210 and IFMS 211 of GMCC 203 with reference to FIG. 2. GRTM 301receives reconfiguration information from IFMS 211 associated with aswitchover in POS interface locations. GRTM 301 then uses thatinformation to update the global route tables in the system with the newlocation of the egress POS interface. GRTM 301 distributes the updatedroute information to all cards within TNR 201 (FIG. 2).

IFMS 306 manages communication and state distribution. When messages arebeing sent to IFMS 306 by client modules (IFMC), they are stamped with asequence number retrieved in each case by an APS CL. This is to insurethat no messages are considered out of order. Distributed APS clientsalso communicate under a timestamp rule. IFMS 306 provides updateinformation via control messaging to APS clients through an IFMCanalogous to IFMC 212 described with reference to FIG. 2.

Each line card (top layer of suite 300 labeled Line) supporting primaryPOS interfaces of an APS group has a primary APS CL (PAC) 302. PAC 302is analogous to APS CL 213 of LC 205 described with reference to FIG. 2.A PAC exists on a line card that has a POS interface designated as aprimary, and supports one or more primary POS interfaces of an APSgroup. A BAC 303, analogous to APS CL 213 of LC 204 described withreference to FIG. 2 only exists on a line card that has a backup POSinterface. On a line card having a backup POS interface and a primaryPOS interface there is a PAC 302 and a BAC 303 residing thereon. WhenPAC 302 detects failures on the primary line, it communicates with theBAC of the same group. Each message sent to IFMC 302 from PAC 302 istimed-stamped using a sequence number.

A wait-to-restore (WTR) timer module 304 is provided to run on a linecard that is configured as a BLC having a backup POS device. Thefunction of WTR module 304 is to provided a time period for a POS backupto wait before initiating a reverse switchover to the primary POSinterface after it is detected that it is up and running again and nolonger requires backup services.

IFMC 305 is analogous to IFMC 212 described with reference to FIG. 2above. IFMC 305 acts as a proxy agent for IFMC 306 on a CC. There is oneIFMC 305 on LC within a TNR router. IFMS acts as a conduit forconfiguration and control information sent from IFMS 306, and also as arelay agent for POS interfaces. It functions to relay their messages toIFMS 306 on a control card. This set of messages includes all state andlocation-related messages.

IFMC 305 is responsible for saving a copy of POS configuration and stateinformation locally so that the information can be fed into a POS deviceon demand. As such, IFMC 305 is responsible for saving state updates ona particular POS device wherein the updates are received from IFMS 306.In addition, IFMC 305 receives configuration data for all APS groupsthat have a primary or backup POS interface residing on that particularhosting line card.

IFMC 305 stores received configuration and state information locally andutilizes the just-received configuration and the previous configuration(if any) to determine if there have been any configuration and/or statechanges. Any changes found are used to update other data structures thatcontain state and configuration information for applicable POSinterfaces for that line card. For example, if a primary line is deletedfrom an APS group, IFMC 305 can delete all state and configurationinformation related to the one (or more) POS interfaces that run on thatline from its data structures. It is noted herein that four SONET pathsmay be part of a single line as was described previously in thisspecification. POS interfaces therefore may be thought of as specific toa path, although when a location changes, all of the paths of thatlocation change.

It will be apparent to one with skill in the art that the modulesillustrated in this example will retain APS protocol standards andrules, but can provided enhanced functions that APS software currentlydoes not provided or support. The innovative concept of APS distributionover multiple processors within a router enables such a router to enjoyAPS/SONET protection by providing a communication and mirroring systemthat performs within the timeframe of 50 ms required for switchover byAPS protocol. FIG. 4 is a block diagram illustrating components of aninterface-configuration-replications-set (ICRS) 400 according to anembodiment of present invention. ICRS 400 comprises all of the state andconfiguration parameters required to completely initiate and activate abackup POS interface to provide backup services for a primary interface.Illustrated within ICRS 400 is a set of data-link layer configurationparameters (LCP CONFIG) 402. Each POS interface has its own LCPconfiguration parameters and 4 POS interfaces (SONET paths) may comprisea SONET line. Also included in ICRS data set 400 is data-link layerstate parameters (LCP STATE) 401. LCP STATE 401 includes updated sateinformation updated as it is received. All LCP data must be mirrored toand reconfigured at a new POS location before switchover to backupservices. Another component of ICRS 400 is SONET line configuration data(SONET LINE CONFIG) 403. This comprises all of the data required toconfigure a primary line in a new location. If a SONET line includes asingle SONET path, the configuration parameters may be the same. Ifthere are 4 SONNET paths to consider, then data comprising SONET pathconfiguration (SONET PATH CONFIG) 404 are included in

ICRS 400. The network protocol layer regards each separate SONET path asa separate physical interface each dedicated to handle a separate dataflow. Therefore, if a SONET line has 4 SONET paths, then it can transmit4 separate data flows simultaneously.

An APS group may be modified on the fly by adding lines to or deletinglines from the group, by changing a backup location, and so on.Therefore, part of the ICRS data includes current APS groupconfiguration data (APS GROUP CONFIG) 406. If an administrative orderincludes adding a SONET line to an APS group, then the ICRS for thatgroup needs to be updated to reflect the modification. If a line iseliminated from a group, all of the associated parameters within theICRS are deleted and new ICRS updates are distributed locally. In thiscase GRTM information must also be updated.

Other data parameters included within ICRS 400 comprise the operatingnetwork protocols and rules of transport operating over the primary linethat is requesting backup. Examples of types of protocols includemulti-layer-switching-protocols (MPLS), ISIS protocols, InternetProtocols (IP) and so on. In one embodiment of the present invention, abackup POS interface may be pre-configured with certain protocols thatare common to all of the interfaces thereby eliminating some of the datawithin an ICRS. ICRS 400 is stored locally on a line card acting as abackup card or BLC so that the data may be fed to a backup POS interfaceon demand.

It will be apparent to one with skill in the art that providing a readyinterface configuration replicated set of data required to activate andoperate a primary interface locally to a backup interface eliminatesmuch work that would otherwise be required to mirror the interface atthe new location.

FIG. 5 is a process flow diagram illustrating steps for initiating anAPS switchover according to an embodiment of the present invention. Atstep 501, a problem is detected on a SONET line belonging to an APSgroup. There are rules that exist for what type of fail signal oradministrative request signal will require backup services. An APS CLrunning on the LC of the failed line detects the failure signal.

At step 502, the APS CL using the IFMC running on the same card sends arequest to an IFMC application running on the BLC supporting the backupinterface of the APS group. The request results in availabilitydetermination of backup services for the failed line. If the backupinterface is busy and priority of the requesting line does not overrideexisting priority activity, then the backup interface of the group isdetermined to be not available. In this case the process terminates atstep 503 and the requesting line will not receive backup services. If atstep 502 it is determined that the backup interface is available thenthe BLC APS notifies the APS primary clients that APS switchover willcommence.

At steps 505 and 506, the primary POS interface needing backup issuspended and the new POS location is initiated for ICRS transfer. Atstep 507, the ICRS used at the primary interface is mirrored to the BLCfor selection and implementation at the backup interface. Step 507 maybe performed well before other steps of this process in an embodimentwherein all of the primary ICRS active in an APS group are mirroredlocally to the BLC for that group. Selection from a table of ICRSs isperformed on the BLC by the IFMC. Meanwhile, the IFMS running on thecontrol card responsible for managing the APS group and supportingcomponents updates the ICRS with any required new data in terms of stateinformation or configuration data.

At step 508, the new POS location is activated according to the ICRSdata and begins a handshake and transmission of data to the othercommunicating party or system. At step 509, the global route table(GRTM) is updated with new route information for data formerly using theprimary that was backed up. The IFMS running on the appropriate controlcard notifies a GRTM running on the same card that an update anddistribution of GRTM data is required.

In a preferred embodiment, a message sequence system is used to ensurethat messages sent from the PLC and BLC are considered in proper orderwhen received at the responsible CC.

It will be apparent to one with skill in the art that the stepsillustrated in this example may be further divided into sub-stepswithout departing from the spirit and scope of the invention. Forexample, steps for performing message checks for errors may beintegrated in to the basic APS messaging process. In another embodiment,a similar process is used to implement a reverse switchover wherein aprimary interface has ceased to function and can resume without backupservices. In still other embodiments, priority schemes are introducedwherein determination and selection steps can be added in case of morethan one primary line requiring backup.

The method and apparatus of the present invention may be practiced overthe Internet network using SONET enhancements as described in thisspecification, or over any WAN adapted to support the required APS andrelated protocols. By providing communication management and reportingAPS components in distributive fashion over multiple processors, anydistributed processor router can be configured for APS switchover insuch a way that a switchover process occurs easily within 50 msproviding added redundancy to an already highly flexible system.

The method and apparatus of the present invention should be afforded thebroadest scope possible under examination. Only the language of thesupported claims, which follow, shall herein limit the spirit and scopeof the invention.

1-11. (canceled)
 12. A distributed processor router, comprising: aplurality of communicating processors supporting a plurality ofcommunication interfaces; an APS server module running on a first one ofthe plurality of processors for managing communication and distributingconfiguration and state information; and APS client modules running onsecond ones of the multiple processors, the APS client modules formonitoring interface state information, reporting to the APS servermodule, and for negotiating with other APS client modules; characterizedin that APS interface relocation from a primary interface to a backupinterface is performed through direct communication between the APSclient modules running on the processors supporting the involvedinterfaces
 13. The distributed processor router of claim 12 wherein thedata-packet-network is the Internet network
 14. The distributedprocessor router of claim 13 wherein the plurality of primary routerinterfaces comprise an APS grouping of interfaces connected to a SONETnetwork.
 15. The distributed processor router of claim 12 wherein theAPS software suite includes a server application, a server-clientapplication, and a client module.
 16. The distributed processor routerof claim 15 wherein the server application runs on a control card, andthe server-client application as well as the client module run on a linecard.
 17. The distributed processor router of claim 12 whereinindication of an event is an APS signal received through the targetinterface on the backup processor.)
 18. The distributed processor routerof claim 17 wherein the received APS signal indicates one of failure orsevere degradation of the target interface.
 19. The distributedprocessor router of claim 17 wherein the received APS signal indicatesan administrative request for interface relocation.
 20. The distributedprocessor router of claim 12 wherein configuration and state informationgeneric to a targeted interface for relocation is mirrored to the backuprouter interface for the purpose of initializing and activating thebackup interface to function as the primary interface.
 21. Thedistributed processor router of claim 12 wherein the distributedprocessors communicate with each other through a network of fabric cardsimplemented within the router. 22-23. (canceled)
 24. A method forrelocating a primary router interface to a designated backup routerinterface using an APS suite distributed over multiple processors of adistributed processor data router comprising steps of: (a) mirroringcurrent configuration and state information of the primary routerinterface to the processor supporting the designated backup routerinterface; (b) receiving indication of a requirement to initiate an APSswitchover; (c) determining if the backup router interface is available;and (d) activating the designated backup interface using the mirroredconfiguration and state information.
 25. The method of claim 24,comprising an additional step (e) for reporting any changed routeresults to a task manager responsible for distributing updated routetables to processors.
 26. The method of claim 25, comprising anadditional step for updating a forwarding data base according to aswitchover made.
 27. The method of claim 24 wherein the distributedprocessor data router is connected to and operating on adata-packet-network at the time of interface relocation.
 28. The methodof claim 27 wherein the data-packet-network is the Internet network. 29.The method of claim 24 when the primary router interface is a part of agroup of interfaces connected to a SONET network.
 30. The method ofclaim 24 wherein in step (b) the indication is received at the primaryinterface.
 31. The method of claim 24 wherein, in step (b), theindication is received at the backup interface.
 32. The method of claim24 wherein in step (b) the indication is of the form of anadministrative request.
 33. The method of claim 24 wherein in step (c)determination of availability of the backup interface partly depends ona priority state of the interface requiring backup. 34-35. (canceled)